1
Modifying the MARS Scale to Evaluate
Health Applications
Session #107, February 13, 2019
Speaker: Ryan Romero, MPH Student, University of Florida
Rick Kates, Clinical Assistant Professor, University of Florida
2
Ryan Romero, MPH Student
Has no real or apparent conflicts of interest to report.
Frederick Kates, PhD
Has no real or apparent conflicts of interest to report.
Conflict of Interest
3
1. Introduction to MARS
– A. Sections of the MARS Scale
– B. Application Specific Measures
– C. Validation of Content Specific Measures
2. Study Specific Populations and Our Application of MARS
– A. The deaf and Hard of Hearing Community
– B. Use of Hard of Hearing Applications
– C. App Specific Measures Developed for Target Population
– D. Study Characteristics
3. New Criteria For Application Specific Measures Using MARS
– A. Identification of a content expert in subject field (with example)
– B. Use of content expert or population member to help develop content specific
measures
– C. Use of content expert to gauge specific interests and needs of population
– D. Short wrap up discussion
Agenda
4
• Design content-specific criteria to evaluate mobile
health applications
• Identify and engage an appropriate content
expert
• Identify population-specific needs and implement
criteria into rating scale
Learning Objectives
5
• Content Expert: Knowledge expert in a particular field of study
that has some professional involvement in the subject
[1]
• Application Specific Measures (Content Specific): Evaluation
metrics designed to gauge the value of specific features towards
an intended population/ function
[2]
• Appropriateness: Relevance of features to intended users
• MARS (Mobile Application Rating Scale): Quantitative (1-5)
scale used to evaluate mHealth applications
[2]
Key Terms
7
1. Health Infinity
8
2. Carb Manager
9
3. MyNetDiary
10
• The Mobile Application Rating Scale (MARS)
[2]
• Developed at Queensland University of
Technology
– Stoyan Stoyanov, Leanne Hides, David Kavanagh,
Oksana Zelenko, Dian Tjondronegoro, Madhavan
Mani
• Provides a standardized approach to comparing
and rating health applications for mobile devices
(iPhone, iPad, Android, Android Tablet)
• Allows for modularity as key part of design
• Frequently used to evaluate mHealth applications
Introduction to MARS
11
Sections of the MARS Scale
[2]
Engagement
5 subsections
Functionality
4 subsections
Aesthetics
3 subsections
Information
7 subsections
Subjective
Quality*
12
Alpha test of interrater agreement is preformed to
ensure acceptable levels of agreement
[3]
.
Sections where there is a lack of agreement or
uncertainty are discussed and resolved by
consultation of the content expert and
reevaluation.
Training and Validation of Raters
13
• MARS allows for custom measures intrinsically
– Tells us about app quality
– Tells us about appropriateness and intended fit
• Use of section as indicator of status vs indicator for change*
– Section can be used to both address quality and declare the
need or desire for specific feature
• Content development and developer feedback
– Use of content expert to guide study’s intent
– Feedback to developers
– Empirical data to support stance
Content Specific Measures
14
Study Specific Populations
and Our Application of the
MARS
15
New Translation Technologies
16
• The Deaf and Hard-of-Hearing community
- 7 million members of HOH community
[6]
• Electronics as part of social interface capability
• Electronics used to communicate with family and friends
• Necessity for evaluation of hard-of-hearing applications
Hard of Hearing Apps & Our Study
17
• Preliminary Literature Review
– 37 million with hearing difficulties
[7]
– Needs of population over 60 years old
[8]
• Decision to Include Content Expert
• Inclusion and Exclusion Criteria
– Need to determine appropriateness
[9]
– Mitigate risk
• Results
Study Characteristics
18
• Application features
• Is technology likely to be used?
• Inclusion and Exclusion Criteria
• Multi category approach to population specific applications
– Text to Speech
– ASL Translator
– Hard-of-Hearing Assistant
Hard of Hearing Applications
19
Content Expert
Stephen J. Hardy, II
M.Ed.
Lecturer, University of Florida
College of Public Health and Health
Professions
Fluent in Deaf Culture, ASL, and Deaf
Studies
20
App-Specific Measure #1
21
App-Specific Measure #2
22
App-Specific Measure #3
[4]
23
App-Specific Measure #4
24
Related Study
[5]
25
• 5 / 21 applications no longer available for evaluation after follow
up
– Simply disappeared
– Indicator of status of application field
– Potential lack of content validity
– Niche product does not necessarily entail less competition
Application Turnover
27
• Three important topics:
1. Use of content experts as guide for project
2. Use of content experts to achieve validity of experiment design
3. Approaching a content expert for consultation
Content Experts and Design Validity
28
• What qualifications should my content expert have?
– How should they relate to the topic we are studying?
• How can they be used to the advantage of the study?
– Guide for purpose
– Who does study effect?
– Discussion section
• Use of multiple content experts
Identification of a Content Expert
29
• Knowledge on use of content experts in
experimental design and analysis
• Recognition of importance of content
validity and designing a study to help the
subject it focuses on
• Recommendation of quality apps based
on MARS
How Can This Information Help YOU?
30
• Discussion on how this information relates to your focus
• Using this information to develop a study
• How health information professionals can use this
information to design better applications
Discussion and Q&A
31
• Ryan Romero, MPH Student
– ryan.romero1000@ufl.edu
• Rick Kates, PhD
– kates.rick@phhp.ufl.edu
Please feel free to come up and speak with us after the presentation!
Speaker Contact
32
1. Grant, J.S., & Davis, L.T. (1997). Selection and use of
content experts in instrument development. Research
in Nursing & Health, 20, 269–274.
2. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M
Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps JMIR
Mhealth Uhealth 2015;3(1):e27. URL: https://mhealth.jmir.org/2015/1/e27
3. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull
19790101;86(2):420. [doi: 10.1037/0033-2909.86.2.420]
4. SkoogMusic. The Skoogmusic Skwitch. Digital Image. Accessed from
http://skoogmusic.com/skwitch/
5. Arellano, Patricia, Bochinski, Janet, Elias, Beth, Houser, Shannon, Martin, Thomas, Head, Hank.
2012. Selecting a Mobile App: Evaluating the Usability of Medical Applications. mHIMSS App
Usability Work Group.
6. Carroll DD, Courtney-Long EA, Stevens AC, Sloan ML, Lullo C, Visser SN, Fox MH, Armour BS,
Campbell VA, Brown DR, Dorn JM. Vital signs: disability and physical activity--United States,
2009-2012. MMWR Morb Mortal Wkly Rep 2014 May 9;63(18):407–413. PMID:24807240
7. Ross, Mitchell, Karchmer, Michael. (2004). Chasing the Mythical Ten Percent: Parental Hearing Status of Deaf and Hard of Hearing Students in the
United States. Sign Language Studies, (4)2; 138-163.
8. Hoffman, H. J., Dobie, R. A., Losonczy, K. G., Themann, C. L., & Flamme, G. A. (2017). Declining Prevalence of Hearing Loss in US Adults Aged 20
to 69 Years. JAMA otolaryngology-- head & neck surgery, 143(3), 274-285.
9. Lewis, T. L., & Wyatt, J. C. (2014). mHealth and mobile medical Apps: a framework to assess risk and promote safer use. Journal of medical Internet research, 16(9), e210.
doi:10.2196/jmir.3133
References
